AD-A203  966 


rr»  rnnv 

ni:  FfiE  nopt 


AFV;AL-TR-C8-1083 


A  METHODOLOGY  FOR  SCREENING  POTENTIAL  ARTIFICIAL 
INTELLIGENCE  APPLICATIONS 


Lt  Eric  N.  Hanson 

Artificial  Intelligence  Applications  Office 


JULY  1988 


PROCEEDINGS 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED 


AVIONICS  LABORATORY 

AIR  FORCE  WRIGHT  AERONAUTICAL  LABORATORIES 
AIR  FORCE  SYSTEMS  COMMAND 

WRIGHT-PATTERSON  AIR  FORCE  BASE,  OHIO  A5A33-65i'43 


DTlC 

electe' 


NOV  2  8  1988 


4 


88  n  £8  04-2. 


NOTICE 


WHEN  GOVERNMENT  DRAWINGS,  SPECIFICATIONS,  OR  OTHER  DATA  ARE 
USED  FOR  ANY  PURPOSE  OTHER  THAN  IN  CONNECTION  WITH  A  DEFINITELY 
GOVERNMENT-RELATED  PROCUREMENT,  THE  UNITED  STATES  GOVERNMENT 
INCURS  NO  RESPONSIBILITY  OR  ANY  OBLIGATION  WHATSOEVER.  THE  FACT 
THAT.  THE  GOVERNMENT  MAY  HAVE  FORMULATED  OR  IN  ANY  WAY  SUPPLIED 
THE  SAID  DRAWINGS,  SPECIFICATIONS,  OR  OTHER  DATA,  IS  NOT  TO  BE 
REGARDED  BY  IMPLICATION,  OR  OTHERWISE  IN  ANY  MANNER  CONSTRUED,  AS 
LICENSING  THE  HOLDER,  OR  ANY  OTHER  PERSON  OR  CORPORATION;  OR  AS 
CONVEYING  ANY  RIGHTS  OR  PERMISSION  TO  MANUFACTURE,  USE,  OR  SELL 
ANY  PATENTED  INVENTION  THAT  MA.Y  IN  ANY  WAY  BE  RELATED  THERETO. 

THIS  REPORT  HAS  BEEN  REVIEWED  BY  THE  OFFICE  OF  PUBLIC 
AFFAIRS  (aSD/cPA)  AND  IS  RELEASABLE  TO  THE  NATIONAL  TECHNICAL 
INFORMATION  SERVICE  (nTIs).  AT  NTIS,  IT  WILL  BE  AVAILABLE  TO  THE 
GENERAL  PUBLIC,  INCLUDING  FOREIGN  NATIONS. 

THIS  TECHNICAL  REPORT  HAS  BEEN  REVIEWED  AND  IS  APPROVED  FOR 
PUBLICATION . 


Chief,  Artificial  Intelligence 
Applications  Office 


FOR  THE  COMMANDER 


ERIC  N.  HANSON,  ILT,  USAF 
Technical  Director 


PATRICIA  L.  SISS*^"',  Lt  Col.  USAF 
Deputy  D i rector 
System  Avionics  Division 
Avionics  Laboratory 


If  your  address  has  changed,  if  you  wish  to  be  removed  from  our  mailing 
list,  or  if  the  addressee  is  no  longer  eiyjloyed  by  your  organization  please 
notify  AFWAL/AAAA,  WPAFB,  OH  45433-_^£f£_  to  help  us  maintain  a  current 
mailing  list. 


Copies  of  this  report  should  not  be  returned  unless  return  is  required  by 
security  considerations,  contractual  obligations,  or  notice  on  a  specific 
document . 


la.  REPORT  SECURITY  CLASSIFICATION 

UnclasBifled 


2a,  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNCRADING  SCHEDULE 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 


fonn  Approved 
OM8  No.  0704-01 $a 


icrinTf 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S)  ^ 

AFVArr-TE-88-1083 

6a.  NAME  OF  PERFORMING  ORGANIZATION 

aFWAL 

6b.  OFFICE  SYMBOL 
(If  applicable) 

AAAA 

6c.  ADDRESS  {City,  State,  and  ZIP  Code) 

1 

WPAFB  OH  A5A33-65A3 

8a.  NAME  OF  FUNDING /SPONSORING 
ORGANIZATION 

8b.  OFFICE  SYMBOL 
(If  applicable) 

See  6a. 

See  6b. 

ac,  ADDRESS  (Ciry,  State,  and  ZIPCode) 

See  6r 

3.  DISTRIBUTION /AVAILABILITY  OF  REPORT 

Approved  for  public  release', 
distribution  unlimited. 


5.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 


Sarae  as  6a. 


7b.  ADDRESS  (Oty,  State,  and  HP  Code) 
Same  as  6a. 


1  10.  SOURCE  OF  FUNDING  NUMBERS 

PROGRAM 

PROJECT 

TASK 

WORK  UN' 

ELEMENT  NO. 

NO, 

NO 

ACCESSION  NO; 

62204F 

2003 

05 

54 

11,  TITLE  (Include  Security  Classification) 

A  Methodoloby  for  Screening  Potential  Artificial  Intelligence  Applications  (Unclassified) 


12.  PERSONAL  AUTHOR(S) 

Eric  N.  Hanson,  Stephen  E.  Cross,  and  Allefi  Smith 


13a.  TYPE  OF  REPORT  13b.  TIME  COVERED  14.  DATE  OF  REPORT  (Year,  Month,  0»y)  15.  PAGE  COUNT 

Proceedings  FROM  —  TO  -  July  1988  20 


1/. 

COSATI  CODES  | 

FIELD 

GROUP 

SUB-GROUP 

— 

■ - 

18.  SUBJECT  TERMS  (Cont/nuf  on  rtverff  if  necf«#fy  and  identify  by  bfock  number) 

12.09 


19.  ABSTRACT  (Conf/nue  on  reverse  if  necessary  and  identify  by  block  number. 

This  paper  presents  a  method  to  rapidly  evaluate  a  large  number  of  possible  opportunities 
to  apply  artificial  intelligence  technology.  The  domain  expert  for  a  potential  applicatioi 
l.a  Interviewed  brief ly ''(for  1-3  hours)  by  a  knowledge-engineering  team.  During  this 
Interview,  the  expert  is  questioned  on  a  carefully  chosen  set  of  topics  to  help  quickly 
assess  the  level  of  AI  risk,  systems  engineering  risk,  and  potential  payoff  of  building 
the  applicatlon(s)  identified  during  the  session.  After  conducting  a  series  of  these 
interviews  for  an  organization,  a  report  is  written  summarizing  the  results,  and  prioritiz¬ 
ing  the  potential  applications  for  future  assessment  and  prototyping.  > 


DO  Form  1473,  JUN  86 


Previous  editions  tre  obsolete. 


« 


UNCLASSIFIED 


»TIC 


% 


A  Methodology  for  Screening  Potential 
Artificial  Intelligence  Applications 


Eric  N.  Hanson 
Stephen  E.  Cross 

Air  Force  Wright  Aeronautical  Laboratories 
Wright- Patterson  AFB,  OH  4^438 


Allen  Smith 


Teknowledge  Federal  Systems,  Inc. 


Abstract 


This  paper  presents  a  method  to  rapidly  evaluate  a  large  number  of  pos¬ 
sible  opportunities  to  apply  artificial  intelligence  technology.  The 
domain  expert  for  a  potential  application  is  interviewed  briefly  (for  1-3 
hours)  by  a  knowledge-engineering  team.  During  the  interview,  the  ex¬ 
pert  is  questioned  on  a  carefully  chosen  set  of  topics  to  help  quickly  as¬ 
sess  the  level  of  A1  risk,  systems  engineering  risk,  and  potential  payoff  of 
building  the  application(s)  identified  during  the  session.  After  conduct¬ 
ing  a  series  of  these  interviews  for  an  organization,  a  report  is  written 
summarizing  the  results,  and  prioritizing  the  potential  applications  for 
future  assessment  and  prototyping. 

1.  Introduction 

A  wide  variety  of  artificial  intelligence  (AI)  applications  have  been  built  successfully 
using  a  knowledge- based  expert  systems  approach.  There  are  examples  in  industry 
lFox84j,  finance  [Cohe83),  medicine  [McDo84],  military  [Ande87]  and  other  fields.  How¬ 
ever,  many  organizations  have  barely  begun  to  explore  the  potential  of  expert  systems. 
This  represents  a  serious  lag  between  technology  and  applications.  Studies  have  shown 
that  it  normally  takes  10-15  years  for  a  new  technology  to  be  widely  applied  unless  spe¬ 
cial  measures  are  taken  to  accelerate  transition  of  the  technology  to  the  field  [Fird88j. 

To  reduce  this  lag  time  in  applying  knowledge-base  systems  technology,  it  is  impor¬ 
tant  to  systematically  identify  potential  applications,  and  carefully  decide  which  ones  to 
pursue  in  order  to  maximize  return  on  investment.  In  a  large  organization,  there  may  be 
literally  dozens  of  opportunities  for  building  knowledge-based  systems.  A  method  is 
needed  to  do  the  following; 
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identify  as  many  of  these  applications  as  possible  to  help  make  sure  the  most 
promising  ones  are  uncovered, 


•  quickly  evaluate  each  application  in  terms  of  risk  and  value, 


prioritize  the  applications  for  future  in-depth  assessment  and  possible  prototyping 
based  on  the  results  of  the  risk /value  analysis, 


•  summarize  the  results  of  the  process  for  management  to  help  them  make  effective 

decisions  about  which  applications  to  pursue. 

This  type  of  approach  contrasts  sharply  with  the  more  conventional  method  of  develop¬ 
ing  AI  systems.  Normally,  potential  AI  applications  are  considered  one  at  a  time,  within 
individual  operational  organizations.  This  grass-roots  approach  makes  it  difficult  for  a 
large  organization  to  maximize  the  organization-wide  payoff  for  a  given  level  of  invest¬ 
ment  in  AI  technology.  The  goal  of  the  approach  described  in  this  paper  is  to  allow  large 
organizations  to  make  the  most  effective  possible  AI  investment  decisions. 

The  application  screening  methodology  discussed  here  was  developed  by  Tek- 
nowledge  Federal  Systems  and  the  Dayton  Center  for  Artificial  Intelligence  Applications 
(CAIA).  The  primary  goal  of  the  CAIA  is  to  help  the  Air  Force  exploit  AI  technology. 
The  Air  Force,  like  any  large  organization,  has  many  potentially  high-payoff  applications 
for  practical  artificial  intelligence  technology;  the  methodology  described  here  was 
developed  to  provide  an  efficient  technique  for  identifying  those  applications.  A  more 
detailed  discussion  of  the  CAIA  is  given  below. 


The  Center  for  Artificial  Intelligence  Applications 


The  CAIA  is  an  independent  organization  sponsored  by  the  Air  Force  Wright 
Aeronautical  Laboratories  (AFWAL)  and  managed  by  the  Miami  Valley  Research  Insti¬ 
tute  (MVRI),  a  consortium  of  universities  in  the  Dayton  area.  Technical  coordination  of 
Center  activities  is  provided  by  Teknowledge  Federal  Systems,  under  a  subcontract  to 
MVRI.  The  Center  has  a  threefold  mission: 

•  to  monitor  and  stimulate  advances  in  the  AI  R&D  community  and  transition 
emerging  AI  technologies  into  Air  Force  applications, 

•  to  identify,  assess,  and  prototype  high-payoff  applications  for  AI  technology  for  the 
Air  Force,  and 


•  to  provide  education  and  training  to  expand  the  pool  of  professionals  with  AI  skills 
in  the  Dayton  area.  The  Center  focuses  on  applying  existing  technology,  stimulat¬ 
ing  and  transitioning  relevant  new  technology,  training  AI  applications  engineers, 
and  improving  the  awareness  of  management  about  the  capabilities  of  AI  technol¬ 
ogy.  The  Center  is  strongly  applications-oriented.  Applied  research  and  develop¬ 
ment  sponsored  by  the  Center  is  driven  by  identified  Air  Force  applications  require¬ 
ments. 

A  key  aspect  of  the  Center’s  charter  is  the  systematic  identification  and  assessment 
of  potential  applications  for  AI  technology  in  the  Air  Force,  to  meet  this  need  for  a  sys¬ 
tematic  and  replicable  screening  methodology,  Teknowledge  Federal  Systems,  working 
with  the  Air  Force  and  MVRI,  developed  the  methods  outlined  in  this  paper.  Using 
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these  methods,  the  Center  identifies  the  application  opportunities  available  to  the  Air 
Force  customer,  analyzes  those  opportunities  in  terms  of  technical  risk  and  potential 
payoff  to  the  Air  Force,  and  recommends  a  plan  for  pursuing  the  most  promising  applica¬ 
tions  further.  The  Center  focuses  primarily  on  relatively  high-payoff  applications.  In 
terms  of  technical  risk,  however,  the  scope  is  broader  and  more  eclectic;  near-term  appli¬ 
cations  of  low-to  moderate  risk  are  encouraged,  but  high-risk  work  is  not  avoided  if  the 
payoffs  are  high  and  risk  can  be  decreased  by  applying  the  results  of  relevant  applied 
research. 

The  rest  of  this  paper  describes  the  method  for  performing  the  steps  in  the  screen¬ 
ing  process.  Section  2  describes  the  methcxlology  in  detail.  Section  3  presents  a  case  study 
where  the  application  was  applied  to  a  particular  organization.  Finally,  Section  4  sum¬ 
marizes  and  presents  conclusions. 

2.  The  Application  Screening  Methodology 

A  detailed  breakdown  of  the  steps  in  the  application  screening  methodology  is  illus¬ 
trated  in  Figure  1.  Each  of  these  steps  is  described  in  more  detail  below. 

2.1.  Prescreening 

Distribute  Initial  Selection  Questionnaire:  To  help  gather  information  about 
potential  AI  applications  in  an  organization,  a  questionnaire  is  distributed  to  key  indivi¬ 
duals.  We  have  found  it  most  effective  to  have  the  management  of  the  organization  give 
the  questionnaire  to  people  they  feel  are  best  qualified  to  complete  it.  The  questionnaire 
we  use  is  shown  in  Appendix  A.  We  feel  that  the  questionnaire  is  a  good  compromise 
between  thoroughness  and  length.  It  captures  the  essence  of  a  potential  application,  but 
it  is  not  so  long  that  people  refuse  to  complete  it. 

Initial  selection  of  problems  to  screen:  The  questionnaires  and  other  informa¬ 
tion  available  about  potential  applications  in  the  organization  being  screened  are 
reviewed.  Obviously  inappropriate  applications  are  culled  out  at  this  point.  Screening 
sessions  are  scheduled  for  the  remaining  applications. 

Pre-brief  of  Participating  Experts  and  Managers:  The  pre-brief  is  a  short 
meeting  where  members  of  the  screening  team  explain  the  screening  process  to  the  organ¬ 
ization  being  screened.  At  the  pre-brief,  the  following  points  are  made: 

•  It  is  very  important  that  the  person  most  familiar  with  the  problem  being  presented 
(the  expert)  is  the  one  who  will  be  interviewed  at  the  screening.  There  is  a 
significant  tendency  of  organizations  to  either  send  the  expert’s  boss  since  the  boss 
is  more  comfortable  with  public  speaking,  or  send  an  apprentice  since  the  expert  is 
too  busy  to  participate.  Working  with  a  manager  who  does  not  know  the  details  or 
an  eager  but  inexperienced  person  will  not  work. 

•  The  people  to  be  screened  are  asked  to  prepare  a  set  of  view-graphs  using  templates 
provided  by  the  screening  team.  The  templates  are  designed  to  be  easy  for  the  ex¬ 
pert  to  complete,  and  still  communicate  the  most  essential  information  to  the 
screening  team.  It  is  important  to  give  the  person(s)  being  screened  a  format  to  fol¬ 
low  since  otherwise  they  tend  to  bring  in  slides  or  other  materials  they  have 
prepared  for  other  presentations,  or  prepare  a  presentation  that  gives  a  general 
overview  of  what  they  do.  These  materials  usually  are  not  very  effective  at  com¬ 
municating  the  nature  of  the  problem  and  the  problem-solving  strategies  used  by 
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Figure  1.  The  Application  Screening  Process: 


the  expert.  An  outline  of  the  templates  used  is  shown  in  Appendix  B. 


2.2.  Screening  Interviews  and  Analysis 

The  Screening  Session:  The  screening  session  itself  is  a  1-1/2  to  3  hour  informal 
interview  with  the  expert.  The  session  is  conducted  by  the  screening  team  which  will  be 
discussed  in  more  detail  in  the  next  section.  The  essential  people  at  the  screening  session 
are  the  expert,  and  the  screening  team  leader.  Other  people  may  be  present  at  the  ses¬ 
sion,  including  additional  screening  team  members,  and  interested  people  associated  with 
the  expert,  possibly  including  co-workers  and  managemejit. 

At  the  beginning  of  the  session,  the  team  leader  introduces  the  team  to  the  expert, 
and  describes  the  format  of  the  session.  The  expert  is  reassured  that  the  session  will  be  a 
friendly  interview  (not  an  inquisition)  and  the  format  will  be  loosely  structured.  The 
template  slides  completed  by  the  expert  will  serve  as  a  guide,  and  there  will  be  frequent 
questions  by  the  team  leader  and  perhaps  other  team  members. 

During  the  session,  the  expert  gives  a  presentation  on  his  or  her  area  of  expertise 
based  on  the  template  slides.  The  team  leader  will  interrupt  as  necessary  to  ask  ques¬ 
tions  to  gather  information  that  will  help  determine  the  AI  application  opportunity  (or 
opportunities)  related  to  the  experts  discussion,  and,  for  each  application  opportunity 
identified,  the  Al  risk,  the  systems  engineering  risk,  and  the  value  of  the  application  if 
successful.  The  questions  asked  are  guided  in  part  by  the  items  on  the  screening  form 
described  below. 

Analysis:  the  Application  Screening  Profile.  Simply  stated,  the  goal  of  an 
application  screening  is  to  evaluate  each  candidate  application  along  two  dimensions  of 
technical  risk  and  one  dimension  of  value,  as  indicated  below:  The  purpose  of  the  screen¬ 
ing  form  is  to  graphically  represent  the  judgements  of  the  screening  team  along  each  of 
the  above  dimensions  of  risk  and  payoff. 

•  Systems  Engineering  Risk:  This  dimension  focuses  on  the  application  system  as 
a  whole,  both  intelligent  and  conventional  modules,  and  evaluates  the  technical 
risks  associated  with  prototyping  and  fielding  that  system.  Dimensions  of  risk  that 
are  evaluated  include: 

•  System  complexity  (number  of  intelligent  vs. 

conventional  modules,  integration  requirements), 

•  System  scalabi'ity  (ability  to  scale  from  prototype 

to  full  system  without  major  redesign, 

•  Performance  requirements, 

•  Hardware  requirements  for  development  and  delivery, 

•  Software  requirements  for  development  and  delivery,  and 

•  Maintainability  (extent  to  which  knowledge  and  data 

in  the  system  will  require  frequent  modification  and 

updating). 

•  AI  Technical  Risk.  Focusing  only  on  the  Al  components  of  the  system,  criteria 
on  this  dimension  evaluate  the  extent  to  which  these  components  challenge  the 
current  state  of  AI  technology.  Dimensions  of  AI  technology  risk  evaluated 
include. 
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•  Type  of  application, 

•  Nature  and  availability  of  expertise 

•  Complexity/difficulty  of  the  task  to  be  performed, 

•  Anticipated  role  of  the  system, 

•  Size/complexity  of  the  knowledge  base, 

•  Applicability  of  current  AI  tools, 

•  Advanced  technology  requirements. 

If  the  application  is  at  or  beyond  the  current  state  of  the  art  in  artificial  intelli¬ 
gence,  or  if  a  piece  of  advanced  technology  would  significantly  improve  the  system, 
there  may  be  an  opportunity  for  the  Center  to  support  a  collateral  Applied  R&D 
Project  targeted  at  the  candidate  application. 

•  Value:  This  dimension  looks  at  the  potential  payoff  from  the  system.  Payoffs  can 
be  economic,  (i.e.  cost  savings),  and  non-economic  (enhanced  survivability  or  mis¬ 
sion  effectiveness).  Another  dimension  of  payoff  is  in  the  generality  of  the  system; 
with  relatively  minor  changes,  the  system  developed  for  the  candidate  application 
may  be  adaptable  to  some  other  problems  in  the  Air  Force.  Value  is  of  course  tem¬ 
pered  by  cost;  although  precise  estimates  are  difficult  to  obtain  during  screenings, 
rough  approximations  can  be  developed  for  the  cost  of  developing  both  the  proto¬ 
type  and  fielded  system. 

The  purpose  of  the  Application  Screening  Profile  is  to  graphically  represent  the 
judgements  of  the  screening  team  along  each  of  the  above  dimensions  of  risk  and  payoff. 
An  actual  copy  of  the  screening  form,  completed  for  the  Fuel  Specimen  Analysis  applica¬ 
tion  (to  be  discussed  later),  is  shown  in  Appendix  C.  One  side  of  the  form  has  slots  to  be 
filled  as  well  as  labeled  scales  to  mark,  and  on  the  other  side  has  instructions  for  marking 
the  scales.  There  is  a  major  scale  for  each  of  the  three  broad  categories  already  identified 
(AI  risk,  systems  engineering  risk,  and  value).  Under  each  of  these  major  categories, 
there  are  several  more  specific  scales.  The  specific  scales  are  used  to  gather  detailed  infor¬ 
mation  that  makes  it  fairly  straightforward  to  make  overall  risk  and  value  assessments 
for  the  major  categories.  A  mark  is  made  on  each  major  scale  that  essentially  summar¬ 
izes  the  specific  items  under  it. 

Judgement  is  required  to  summarize  the  marks  on  the  detailed  scales  to  come  up 
with  an  assessments  for  the  major  categories.  It  is  not  wise  to  apply  a  simple  formula 
such  a«  taking  thp  maximum  c>f  the  detailed  scale  values  to  come  up  with  the  value  for 
the  major  scale.  If  one  of  the  detailed  items  indicates  high  risk,  and  the  screening  team 
feels  that  that  item  is  a  "show  stopper"  (i.e.  its  effect  dominates  the  effects  of  the  other 
items)  then  it  is  appropriate  to  mark  the  major  scale  at  the  same  point.  However,  if  the 
high  risk  indication  for  a  particular  item  does  not  dominate  the  others,  it  is  appropriate 
to  average  the  detailed  scale  marks  to  generate  the  mark  on  the  major  scale. 


2.3.  Report  Preparation 

The  Screening  Report:  This  report  summarizes  the  results  of  the  assessment. 
Normally,  many  potential  applications  within  a  large  organization  are  screened.  A  com¬ 
plete  report  is  created  that  discusses  all  the  application  areas  screened  within  the  organi¬ 
zation.  The  report  has  the  following  format: 
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•  Introduction 

•  Description  of  Application  Screening  Procedures  and  Criteria 

•  One  section  for  each  Application  Area 

Each  application  area  section  contains  a  logical  grouping  of  one  ore  more  potential  appli¬ 
cations  There  is  one  subsection  for  each  potential  application.  The  format  of  these  sub¬ 
sections  is  as  follows. 

•  Opportunity:  (title  of  the  potential  application) 

•  Description 

•  Recommendation 

•  Discussion 

•  Overall  Conclusions  and  Recommendations 

•  Appendices 

•  Screening  Profiles  (from  the  screening  forms) 

•  Suggested  Advanced  Technology  R&D  Areas 

•  Application  Screening  Summary 

A  few  sections  of  this  report  merit  more  discussion.  For  a  particular  application  oppor¬ 
tunity,  the  Description  section  normally  contains  a  paragraph  that  concisely  describes  the 
proposed  application  The  Recommendation  section  contains  one  of  the  following  possi¬ 
ble  recommendations  (or  a  customized  recommendation  if  the  ones  below  are  not 
appropriate): 

1  Pursue 

2.  Pursue  with  constraints 

3.  Pursue  (guarded) 

4.  Applied  R&D  Only 

5.  Re-examine  Later 
6  Do  Not  Pursue 

pursue  (1)  is  used  for  those  applications  that  clearly  have  high  value  and  relatively  low 
risk,  pursue  with  constraints  (2)  is  used  for  applications  that  have  high  potential  value, 
but  will  not  be  successful  unless  some  conditions  are  met  (for  example,  a  database  must 
be  built  first,  or  the  one  available  expert  must  agree  to  postpone  retirement  for  six 
months  while  the  system  is  built),  pursue  (guarded)  (3)  is  for  applications  that  seem 
promising,  but  have  a  higher  ratio  of  risk  to  value  than  for  (1)  and  (2),  Applied  R&D 
Only  (4)  is  for  high  value  applications  which  would  be  difificult  or  impossible  to  build 
with  current  technology.  This  recommendation  indicates  a  problem  area  where  applied 
research  is  needed.  Re-examine  Later  (5)  indicates  an  application  that  would  not  be  wise 
to  pursue  now  (for  example,  due  to  lack  of  operational  expertise)  but  which  may  be 
worthwhile  to  pursue  in  the  future  (e.g.  once  enough  expertise  has  been  gathered).  Do 
Not  Purse  (6)  is  for  applications  that  have  a  high  risk-to-value  ratio. 

Making  the  cut:  Invariably,  after  screening  a  large  organization,  building  all  the  poten¬ 
tial  applications  with  “pursue”  recommendations  would  require  more  money  and  man¬ 
power  than  can  be  spared.  Hence,  some  subset  of  the  applications  must  be  selected  for 
development.  Clearly,  the  goal  should  be  to  maximize  the  expected  return  on 
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investment  The  problem  is  complicated  by  the  fact  that  there  are  many  possible  types 
of  “return,”  including 

•  financial  payoff, 

•  incr'  A'^ed  performance  (e.g.  improved  sortie  rate), 

•  in;. eased  acceptance  by  top  management  etc. 

Deciding  which  applications  to  pursue  is  complicated  by  the  fact  that  the  those  with  the 
highest  potential  payoff  also  have  the  highest  risk,  or  require  the  most  development 
effort  We  address  these  concerns  using  the  following  strategy: 

1  Select  one  or  more  low-risk  applications  for  early  development  to  foster  support 
from  management  in  the  organization  as  early  as  possible.  These  should  be  the 
highest-payoff  low  risk  application.®  available. 

2  Select  applications  from  those  that  remain  to  give  maximum  expected  return  on  in¬ 
vestment  This  may  involve  applications  that  are  high  risk,  or  will  be  expensive  to 
develop 

The  first  step  toward  building  an  application  is  to  do  an  in-depth  assessment  to  find 
whether  the  application  is  feasible,  and  determine  how  much  manpower,  time,  hardware 
and  software  resources  will  be  required  The  application  will  be  prototyped  only  if  it  still 
is  favorable  in  terms  of  risk  and  potential  payoff  after  the  assessment.  A  full  discussion 
of  the  methods  used  to  do  the  assessment  and  prototype  are  beyond  the  scope  of  this 
paper  In  the  next  section,  we  give  a  case  study  of  applying  the  screening  methodology 
in  a  large  organization. 


3.  Applying  the  Methodology:  A  Case  Study 

This  section  presents  an  actual  case  study  of  applying  the  screening  methodology 
to  two  US  Air  Force  organizations,  the  Propulsion  Laboratory,  which  is  a  part  of  the  Air 
Force  Wright  Aeronautical  Laboratories  (AFWAL),  and  a  Program  Office  in  Aeronautical 
Systems  Division  (ASD/SC).  Both  are  located  at  Wright-Patterson  AFB,  Ohio,  A  brief 
overview  of  the  application  areas  presented  will  be  given,  followed  by  a  more  detailed  dis¬ 
cussion  of  an  individual  application  opportunity,  Fuel  Specimen  Analysis.  To  begin  the 
screening  process,  representatives  from  the  Propulsion  Laboratory  were  briefed  one  week 
m  advance  to  help  them  prepare  for  the  sessions.  The  session  format  was  explained  to 
them,  and  they  were  given  templates  of  slides  which  they  were  asked  to  have  the  domain 
experts  prepare  prior  to  the  sessions.  The  sessions  themselves  required  three  work  days 
(January  12-14,  1988)  A  summary  of  the  results  of  the  ASD/SD  and  Propulsion  Lab 
report  appears  in  the  table  in  Appendix  D.  The  table  is  designed  to  quickly  identify  the 
applications  opportunities,  identify  the  AI  risk,  systems  engineering  risk,  and  value  using 
graphical  icons,  show  the  recommendation  for  each  application,  and  list  areas  where 
advanced  research  and  development  may  be  needed.  The  portion  of  the  final  report  for 
the  Fuels  Analysis  problem  area  is  shown  in  Appendix  E. 

4.  Conclusions 

The  beauty  of  the  application  screening  methodology  presented  in  this  paper  is  that 
it  allows  a  large  number  of  possible  AI  applications  to  be  identified  and  prioritized  in  a 
short  time.  The  screening  process  provides  a  good  preliminary  picture  of  the  risk  and 
value  of  each  potential  application  through  an  interview  process  guided  by  the  use  of  an 
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appltraliott  acreetiitig  profile  The  profile  contains  a  carefully  selected  set  of  questions 
designed  to  help  derive  good  estimates  of  the  AI  risk,  systems  engineering  risk,  and 
potential  value  of  each  application  opportunity.  The  results  of  screening  many  problems 
m  a  large  organization  are  summarized  in  a  report.  A  key  feature  of  this  report  is  a 
single-page  chart  summarizing  the  results  The  contents  of  the  report  provide  highly 
valuable  guidance  to  the  management  of  the  organization  screened,  helping  them  focus 
their  AI  resources  on  the  best  applications. 

The  screening  methodology  presented  in  this  paper  is  not  a  substitute  for  in-depth 
assessment.  It  is  designed  to  give  a  good  “first  cut”  at  a  large  number  of  possible  appli¬ 
cations.  Only  the  most  promising  applications  identified  during  the  screenings  will  be 
pursued  further  through  in-depth  assessment  and  prototyping. 

The  methodology  presented  here  has  been  successfully  applied  on  a  large  scale  in  the 
Aeronautical  Systems  Division  (ASD)  of  Air  Force  Systems  Command.  Over  200  oppor¬ 
tunities  for  applying  knowledge-based  systems  technology  have  been  identified,  including 
at  least  one  likely  to  provide  multi-million  dollar  pay-off  on  the  funds  used  for  develop¬ 
ment.  The  experience  at  ASD  indicates  that  the  AI  application  screening  process  is  a 
highly  effective  tool.  Strategic  planners  in  other  organizations  would  be  well-advised  to 
adopt  tlie  methodology  for  their  organizations  to  keep  their  competitive  edge. 


Referencea 

[Ande87j 

Anderson,  D  and  C.  Ortiz,  “AALPS:  A  Knowledge  Based  System  for  Aircraft 
l/oading,"  IEEE  Expert,  Winter  1987,  pp.  71-79. 

[Cohe83l 

Cohen,  P.,  and  M.  D.  Lieberman,  “A  Report  On  Folio:  An  Expert  Assistant  for 
Portfolio  Managers,”  Proceedings  of  IJCAI,  1983,  pp.  212-14. 

iFirdbSj 

Firdman,  H  E.  Expert  Systems:  Are  You  Already  Behind?  Computer  World,  April 
18, 1988 

[FoxSlj 

Fox,  Mark  S.  et  al.,  “ISIS:  A  Knowledge- Based  System  for  Factory  Scheduling,” 
Expert  Systems,  Vol  I,  No.  1,  1984. 

[McDo84{ 

McDonald,  Clement  J.,  et  al.,  “CARE:  A  Real  World  Medical  Knowledge  Base,” 
Proceedings  of  CompCon,  Spring  1984,  PP  187-91. 

Appendix  A:  Problem  description  questionnaire 

1.  What  is  the  application? 

a.  Describe  the  application  in  a  nutshell.  What  benefit  does  it  offer  or  what  problem 
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docs  il  solve'’ 


b  W  hat  is  the  subject  matter  area  (general  and  specific)? 

<•  W'liat  exactly  is  the  task  to  be  performed'^  In  a  paragraph,  what  would  the  system 
do'’ 

d.  Why  is  the  application  important?  How  does  it  impact  the  organization? 

e.  What  are  the  inputs  to  the  application  (e.g.  sensors,  test  results,  data  in  databases, 
questions  from  novices  seeking  advice,  poorly  formed  queries  of  users  seeking  data,  a  pro¬ 
posal  to  be  critiqued)? 

f.  What  are  the  outputs  of  the  application  (e.g.  a  diagnosis,  a  plan  for  performing  a  task, 
a  precise  database  query  (or  collection  of  queries),  an  engineering  design  or  diagram,  a 
document  )'f’ 

g.  How  does  this  application  fit  into  the  larger  picture  of  the  organizations  operation? 

h.  If  the  system  is  successful,  what  will  be  the  payoff  for  the  organization? 

i.  How  soon  is  the  application  needed?  Are  there  time  constraints’ 

j.  W’hat  approaches  have  been  tried  in  the  past  to  solve  the  problem'’ 

2  W’hat  issues  must  be  faced  by  the  organization  to  field  a  working  system? 

a  does  the  system  have  to  be  integrated  with  existing  hardware  or  software? 

b  Will  a  change  in  existing  procedures  or  policy  be  required  to  use  the  system?  How 
extensive  a  change?  c.  What  will  be  the  consequences  if  the  system  fails  to  perform? 

d.  How  receptive  and/or  adaptable  to  new  technology  are  the  people  who  will  use  the 
system? 

e  W'hat  kind  of  financial  constraints  exist? 
f.  Any  other  constraints? 

Appendix  B 

Outline  of  slide  templates  for  application  overview  briefing  to  be  given  by  application 
expert  (one  slide  for  each  of  items  1-7). 

1.  Application  Characterization 

•  Application  Overview 

•  Role  of  the  Application 
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2.  Application  Situations 

•  <-\:iiMplc.s  Df  .situat  ions  the  applii  at ion  Would  be  expected  to  handle 

3.  Overall  Flow 

•  Outline  of  the  flow  of  the  expert’s  decision-making  process 

4.  Expertise 

•  Types  and  sources  of  expertise  that  can  be  applied  to  the  problem. 

5.  Example  Case:  Circumstances 

•  Circumstances  surrounding  the  specific  example  case  (problem-solving 
situation).  Hints:  What  are  the  symptoms?  What  relevant  past  histo¬ 
ry,  if  any,  is  available  regarding  this  case  and  its  environment? 

6.  Example  Case:  Step-By-Step 

•  Detailed  breakdown  of  steps  in  solution  of  example  case.  For  each 
step,  this  includes 

•  step  description 

•  input  information  desired 

•  knowledge  used  in  selecting  action 

•  action 

•  result 

7.  Example  Case:  Variance 

•  The  expert’s  analysis  of  how  the  case  was  handled  when  it  actually  oc¬ 
curred.  Hints:  If  you  handled  the  case,  would  you  handle  it  the  same 
way  as  today?  If  you  didn’t  handle  the  case,  would  you  address  it 
differently? 
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APPLICATION  SCREENING  PROFILE 


Appendix  C: 

Application  Screening  Profile  for  Fuel  Specimen  Analysis 
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»  tync  ^ 


Appendix  C  ctd.  (reverse  side  of  application  screening  profile) 
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Appendix  E:  Final  Report  for  Fuels  Specimen  Analysis 
Opportunity  FT-1:  Fuel  Specimen  Analysis 
Description 

This  opportunity  was  described  by  Lt.  •  oward.  Whenever  an  aircraft  mishap 
cx-curs,  whether  military  or  commercial,  specimens  of  the  j  fuel  involved  in  the  mishap 
are  usually  collected  and  sent  to  the  Fuels  Branch  for  analysis,  the  objective  of  the 
specimen  analysis  is  to  detect  and  identify  any  connection  between  the  fuel  and  the 
mishap.  In  addition  to  the  fuel  specimen,  the  analyst  may  receive  a  background  report 
on  the  m;ishap,  including  the  maintenance  record  and  refueling  history  of  the  aircraft. 
Mishaps  include  crashes,  fuel  system  shut-down  on  takeoff,  flameout  at  altitude,  prema¬ 
ture  combustor  line  burn-through,  and  any  other  event  not  having  a  readily  apparent 
proximate  cause,  e  g.,  a  ground  fire,  the  Fuels  Branch  may  be  asked  to  perform  as  many 
as  60  analyses  per  year. 

In  performing  the  analysis,  the  first  step  is  to  select  the  tests  to  be  performed. 
Tests  are  selected  to  verify  some  hypothesis  or  to  resolve  an  ambiguity  among 
hypotheses.  Test  sequencing  is  determined  by  the  cost  of  performing  the  test,  in  terms  of 
both  its  destructiveness  and  the  time  required  for  its  performance.  After  completion  of  a 
battery  of  tests,  results  are  interpreted  and  reported.  If  results  are  ambiguous,  new  tests 
may  be  conducted  to  resolve  the  ambiguity. 

The  planning  of  tests  is  a  key  task  in  the  process,  and  it  is  heavily  dependent  on 
expertise.  An  expert  analyst  uses  knowledge  available  from  previous  incidents, 
knowledge  of  the  operating  features  of  particular  aircraft,  engines,  and  fuel  systems,  and 
knowledge  of  effects  of  environmental  conditions  and  contamination  to  select  and 
sequence  the  most  relevant  tests.  Unfortunately,  expertise  is  limited  and  experts  are 
often  not  available  to  plan  an  analysis.  When  experts  are  not  available,  inappropriate 
tests  may  be  conducted,  often  those  prescribed  by  the  requestor,  which  lead  to  unneces¬ 
sary  testing  and  inconclusive  results.  The  problem  is  exacerbated  by  a  tendency  to 
assume  that  fuels  are  the  causes  rather  than  hardware  or  other  material,  problems  that 
may  be  more  difficult  to  correct.  This  tendency  generates  many  repetitive,  unnecessary 
tests  which  require  large  amounts  of  expert  time. 

Lt.  Howard  proposed  an  expert  system  that  would  provide  advice  to  an  analyst  on 
test  selection  and  sequencing  and  on  interpretation  of  results.  The  system  would  capture 
the  expertise  of  existing  experts  and  would  serve  as  a  surrogate  for  the  expert  in  routine 
analyses,  which  represent  the  majority  of  cases,  this  would  free  the  experts  to  concen¬ 
trate  on  the  difficult,  high-  priority  problems  while  decreasing  the  average  time  to  perfor¬ 
mance  analyses. 

Recommendation 

Pursue  (unqualified).  The  screening  team  feels  that  this  application  opportunity 
may  be  well  suited  to  an  intelligent  systems  approach  and  recommends  that  it  be  pur¬ 
sued  further  with  a  full-scale  assessment. 
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Discussion 


AI  Risk  < - X . > 

SE  Risk  <— X . - . > 

Value  &  Cost  < . X . > 

Low  Moderate  High 


Key  knowledge,  software  and  systems  engineering  factors  for  this  application  opportunity 
include: 

•  AI  Risk 

1  Type  of  Application.  The  application  involves  assess/choose  (diagnosis  of  the 
situation),  plan/choose  (select  tests),  and  plan/construct  (sequence  selected  repairs). 

2.  Satxire  and  Availability  of  Expertise.  Expertise  is  expressible  and  available. 
Experts  specialize,  but  there  is  a  representative  of  the  entire  domain. 

3.  Complexity/ Difficulty  of  Task.  The  task  is  straightforward  for  an  expert. 

4.  Role  of  the  System.  The  proposed  system  would  advise  a  non-expert  analyst. 

5.  Size/  Complexity  of  KnowledgeBase.  The  knowledge  base  is  expected  to  be  of 
moderate  size.  Complexity  is  estimated  as  low  to  moderate. 

6.  Applicability  of  Current  AI  Tools.  Current  commercially  available  tools  will  be 
adequate  for  this  application. 

7.  Advanced  Technology  Requirements.  None. 

•  Systems  Engineering  Risk 

1.  System  Complexity.  The  system  will  consist  of  a  single  intelligent  module. 

2.  System  Scalability.  Low  to  moderate  augmentation  of  the  intelligent  system  is 
expected  in  moving  from  a  prototype  system  to  a  full  system. 

3.  Performance  Requirements.  No  performance  requirements  are  specified. 

4.  Hardware  Requirements.  No  hardware  is  specified. 

5.  Software  Requirements.  No  software  is  specified. 

6.  Maintainability.  System  maintenance  should  be  low  to  moderate. 
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•  Value  &  Cost 


1.  Economic  Value.  The  economic  value  is  expected  to  be  moderate,  attributable 
to  costs  savings  attributable  to  using  less-skilled  analysts. 

2.  Effectiveness.  The  system  should  have  a  moderate  impact  on  effectiveness,  due 
faster,  more  relevant  analyses. 

4  3.  Generality.  Generality  is  moderate  because  the  concepts  and  general  approach 

should  be  transferable  to  other  diagnosis/decision-support  systems. 

*  A.  Cost.  The  cost  of  developing  a  prototype  and  full  system  is  low:  about  one 

person-year. 
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